avcodec/decode: guard against NULL hw_frames_ctx
authorDmitry Rogozhkin <dmitry.v.rogozhkin-at-intel.com@ffmpeg.org>
Fri, 17 Nov 2023 17:03:03 +0000 (09:03 -0800)
committerSebastian Ramacher <sramacher@debian.org>
Sun, 19 Nov 2023 09:48:05 +0000 (10:48 +0100)
commit4d136c8fab6cce5f35b41733855870363ce69198
treeb6f08a5a4ec43a5ed2eb9695d4d2f73aa60d71a6
parent0800c11a259278c2fabe54bae3ef0d1f9ec6650d
avcodec/decode: guard against NULL hw_frames_ctx

Guard against segfault running VLC decoding under msys2 [1]:

Thread 33 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 37728.0xadd0]
ff_hwaccel_frame_priv_alloc (avctx=0x6447b00, hwaccel_picture_private=0x65dfd00)
    at libavcodec/decode.c:1848
1848        frames_ctx = (AVHWFramesContext *)avctx->hw_frames_ctx->data;
(gdb) bt
    at libavcodec/decode.c:1848
    at libavcodec/h264_slice.c:208
    first_slice=1) at libavcodec/h264_slice.c:1599
    at libavcodec/h264_slice.c:2130
    at libavcodec/h264dec.c:652
    got_frame=0x646e4b0, avpkt=0x64522c0) at libavcodec/h264dec.c:1048

(gdb) p avctx
$1 = (AVCodecContext *) 0x6447b00
(gdb) p avctx->hw_frames_ctx
$2 = (AVBufferRef *) 0x0

v2: check for free_frame_priv (Hendrik)
v3: return EINVAL (Christoph Reiter)

See[1]: https://github.com/msys2/MINGW-packages/pull/19050
Fixes: be07145109 ("avcodec: add AVHWAccel.free_frame_priv callback")
CC: Lynne <dev@lynne.ee>
CC: Christoph Reiter <reiter.christoph@gmail.com>
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Gbp-Pq: Name 0002-avcodec-decode-guard-against-NULL-hw_frames_ctx.patch
libavcodec/decode.c